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INTRODUCTION 

When  a  cancerous  lump  is  detected  early  in  the  breast,  the  patient  may  elect  to 
undergo  the  less  traumatic  treatment  of  lumpectomy.  The  treatment  involves  removing 
the  cancerous  lesion  while  leaving  most  of  the  breast  intact.  Aithough  the  surfaces  of  the 
excised  tissue  are  normally  checked  for  signs  of  cancer  to  ensure  that  the  lesion  was 
completeiy  removed,  approximately  1  in  5  of  these  patients  suffer  from  recurrence  of  the 
disease.  Therefore,  lumpectomy  alone  frequently  does  not  render  the  patient  disease 
free.  We  hypothesize  that  diseased  ceils  exist  outside  the  histologically  identifiable 
border  of  the  biopsied  lesion,  most  likely  in  the  form  of  individual  or  small  groups  of  cells 
that  have  extended  from  the  primary  lesion.  These  cells  could  eventually  create  new 
secondary  cancer  foci.  The  alternative  hypothesis  is  that  the  disease  is  really 
multicentric  and  exists  as  a  system  of  independent  non-connected  (neither  physically  nor 
genetically)  foci.  This  project  uses  3D  digital  microscopy  for  analyzing  tissue  structure  at 
multiple  scales  and  in  situ  genetic  analysis  to  recognize  normal  from  diseased  cells  on 
an  individual  basis.  By  combining  these  two  techniques,  we  will  be  able  to  measure  the 
spatial  distribution  of  genetically  aberrant  cells  versus  normal  appearing  cells  beyond  the 
leading  edge  of  the  intraductal  lesions.  These  cancer  cells  lying  in  the  lumen  of 
morphologically  normal  ducts  could  be  easily  overlooked,  using  traditional  histology 
staining.  The  distribution  of  cancer  cells  will  help  us  understand  the  spreading 
mechanism.  Answering  this  question  may  help  us  to  predict  before  surgery  which 
patients  will  suffer  recurrence,  which  patients  need  additional  treatment  following 
lumpectomy  to  avoid  recurrence,  and  provide  valuable  information  in  the  search  for  new 
treatments. 

In  this  project  we  will  use  computerized  microscopy  to  locate  a  lesion  inside  a 
duct  and  to  trace  in  3  dimensions  the  ducts  branching  from  it.  Then,  using  the  technique 
described  above  we  will  detect  the  abnormal  cells  in  the  normal  looking  ducts  extending 
from  the  lesion.  Analysis  of  the  spatial  pattern  of  abnormal  cells  will  tell  us  if  isolated  or 
small  groups  of  cancer  cells  are  present  or  if  some  other  spreading  mechanism  is  taking 
place,  and  how  far  the  spreading  is  from  the  lesion. 
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Our  accomplishments  during  the  first  year  (08/01/00-07/31/01)  of  the  project  will  be 
described  following  the  Tasks  enumerated  in  the  approved  proposal.  The  tasks  are  listed 
below,  and  the  sub-tasks  corresponding  to  the  first  funding  year  have  been  underlined. 
The  text  has  references  to  the  numbered  tasks  (e.g.  see  Task  1.1)  where  the  fulfillment 
of  the  tasks  is  explained.  Before  starting  describing  them,  it  has  to  be  noted  that  most  of 
the  goals  have  been  amply  fulfilled,  including  some  that  were  scheduled  for  the  second 
year,  even  though  the  effective  start  date  of  the  work  was  10/01/01 ,  i.e.  two  months  after 
the  administrative  start  date.  This  has  been  possible  in  part  thanks  to  the  liberality  of  the 
contract  officers,  which  allowed  us  to  join  the  budgetary  efforts  of  this  and  our  other 
grant,  “Three-dimensional  computer-based  mammary  gland  reconstruction  for 
measurement  of  the  patterns  of  hormone  receptor  expression  during  mammary 
developmenf  (DAMD1 7-00-1 -0306),  which  goals  for  the  first  year  partially  overlapped 
with  the  goals  of  this  grant. 

Task  1.  (Months  1-12)  Modify  an  existing  microscopic  imaging  system  for  acquiring  low 
magnification  (1  pixei=  5  pm)  images  of  entire  tissue  sections  and  for  tracing  in  3D 
the  ducts  in  the  tissue  specimen  from  a  series  of  images  of  adjacent  sections. 

1.  Complete  the  existing  JAVA  based  software  for  interactive  marking  and  3D 
virtual  rendering  of  ducts  so  that  it  allows  anv  branching  pattern.  (Months  1-6) 

2.  Interface  the  existing  acquisition  and  registration  software  with  the  JAVA 
aoDlication  to  allow  revisiting  of  acquired  slides  for  inspection  and  hiah-resolution 
acquisition  of  areas  of  interest.  (Months  6-12) 


ACOMPLISHMENTS 


Under  the  combined  budgetary  efforts  of  this  and  grant  DAMD1 7-00-1 -0306,  we  have 
developed  a  client-server  application,  following  the  scheme  shown  in  Figure  1.  The 
server  is  a  C  application  that  runs  on  a  computer  (Dell  Inspiron,  running  Solaris  7  for 
Intel)  physically  attached  to  an  Axioplan  (Zeiss  Inc.,  Germany)  microscope.  The  server 
provides  access  to  the  drivers  that  control  the  moving  parts  of  the  microscope:  motorized 
scanning  stage,  excitation  filter  wheel  and  arc-lamp  light  blocking  shutter  (LudI 
Electronic  Products  Ltd.,  Hawthorne,  NY),  as  well  as  to  the  controller  of  the  CCD 
Microimager  camera  (Xillix  Techonologies  Corp.,  Richmond,  British  Columbia,  Canada). 
Besides  giving  access  to  basic  operations,  such  as  acquiring  and  storing  images,  setting 
the  exposure  time  of  the  CCD  or  moving  the  stage  and  or  the  filter  wheel,  the  server  has 
been  programmed  to  offer  more  complex  functions.  Some  of  them  are  automatic 
focusing  of  the  microscope  or  acquisition  of  multiple-field  of  view  images.  To  do  this,  the 
server  receives  the  order  and  divides  it  into  a  set  of  simple  actions.  For  instance,,  to 
acquire  a  multiple  field-of-view  images,  the  server  takes  the  coordinates  of  the  vertices 
of  the  area  to  acquire  and  then  automatically  performs  the  sequence  of  stage 
movements  and  camera  operations  for  first  creating  a  focus  map  of  the  area  and  then 
acquiring  a  set  of  consecutive  images  that  covers  the  area.  The  output  is  a  mosaic-like 
image  of  the  area.  The  server  permits  multi-color  acquisition  in  fluorescence,  by 
performing  consecutive  acquisition  using  different  excitation  filters. 

The  client  (insofar  referred  as  R3D2)  is  connected  to  the  server  through  UNIX 
sockets,  which  are  the  standard  for  Internet-based  communications.  This  means  that  it 
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can  send  requests  to  the  server  from  any  computer  internet-connected  to  the  server.  To 
obtain  the  maximum  benefit  from  this  fact,  R3D2  has  been  written  in  JAVA  (v.1.2),  so 
that  it  can  be  executed  on  different  computer  platforms. 

Figure  2  shows  the  R3D2’s  Graphical  User  Interface  (GUI).  The  Interface  Is  divided 
in  two  distinguishable  parts.  One  -rightmost  vertical  panel  in  Figure  2-  provides 
connections  to  the  server  and  allows  the  user  to  request  its  services  through  a  user- 
friendly  interface.  The  available  actions  are; 

•  Connect:  Establishes  a  new  connection  with  the  server  via  an  Internet 

socket.  All  the  following  options  assume  that  this  action  has  been 
successful. 

•  Set  Objective  lens:  Our  system  does  not  have  a  motorized  objective  tower, 

and  therefore  this  option  does  not  effectively  change  the  objective  lens 
positioned  in  the  light  path.  However,  the  user  must  update  the  lens  when 
manually  changing  the  lenses,  because  this  information  is  required  for 
some  of  the  remaining  options. 

•  Change  Filter:  Changes  the  excitation  filter.  This  is  needed  when  using 

fluorescence  microscopy. 

•  Exposure  Time:  Sets  the  exposure  time  for  the  CCD  camera  (in  ms.) 

•  Set  Origin:  This  option  moves  the  scanning  stage  of  the  microscope  to  its 

absolute  mechanical  origin  of  coordinates,  relative  to  which  all  future 
movements  and  measurements  are  done.  This  is  required  to  precisely 
locate  the  areas  of  the  slides  where  images  are  taken,  which  is  necessary 
for  accurate  revisiting  of  the  slide  and  correct  reconstruction  of  the  tissue. 

•  Acquire  image:  Takes  an  image  using  the  existing  settings.  R3D2  receives 
the  image  from  the  server  and  displays  it,  both  complete  —zoomed  out-  as 
well  as  partially  -in  their  original  resolution-.  Only  a  small  part  of  the  full- 
resolution  image  can  be  displayed  at  full  resolution.  The  area  can  be 
interactively  determined  by  moving  a  window  on  the  complete  version  of 
the  image  (Figure  3).  Images  can  be  saved  both  in  ICS  [Dean  90]  and 
JPEG  format.  When  images  are  saved  as  ICS,  all  the  acquisition 
parameters  (objective,  filter,  location  of  the  image  on  the  slide  are,  etc.) 
are  stored  in  the  ICS  header  file. 

•  Autofocus:  Automatically  focuses  the  microscope  by  taking  a  series  of 

images  at  different  positions  in  the  Z  axis  (step  size=0.5  microns)  and 
determining  the  best  focused  image  of  the  series.  Blur  due  out  of  focus 
light  reduces  image  contrast,  which  can  be  detected  using  several 
functions.  Based  on  several  comparisons  described  in  the  existing 
literature  [Groen  85][Firestone  91][Santos  97],  we  selected  an 
autocorrelation-based  measurement  introduced  by  Vollath  [Vollath  87]. 
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•  Open/Close  Shutter:  Closes  the  arc  lamp  light  blocking  shutter,  to  reduce 

photobleaching  due  to  unnecessary  light  exposure.  This  is  useful  when 
using  fluorescence  microscopy. 

•  Scan;  Acquires  multiple  field  of  view  images.  The  system  displays  a 

dialogue-panel  (Figure  4)  where  the  user  can  specify  the  filter(s)  to  be 
used  (in  fluorescence  microscpy),  exposure  time(s)  and  the  limits  of  the 
area  to  acquire.  The  limits  can  be  defined  by  its  coordinates  (when 
known)  or  manually,  by  moving  the  microscope  to  the  upper,  lower, 
rightmost  and  leftmost  points  of  the  area.  Figure  5  shows  an  example  of 
an  image  of  a  complete  tissue  section. 

•  Revisit  Point:  When  the  user  clicks  on  a  point  of  a  previously  imaged 

section,  the  server  moves  the  stage  to  that  location  on  the  slide  and  takes 
an  image  using  the  current  values  of  objective,  filter  and  exposure  time. 

•  Revisit  Area:  When  using  the  option,  the  user  is  prompted  to  draw  a 

rectangle  on  the  image  of  a  section.  Then  a  dialogue  panel  similar  to  the 
one  used  to  acquire  sections  asks  the  user  for  the  parameters  of  the 
acquisition(s).  Then  the  system  automatically  acquires  the  area(s).  This 
can  be  used  to  re-acquire  areas  using  a  different  magnification  and/or 
filter  (when  using  fluorescence).  This  feature  greatly  simplifies  some 
tasks,  such  as  determining  and  acquiring  areas  of  interest  (e.g.  sections 
of  mammary  ducts,  blood  vessels,  tumor  micro-invasions,  etc.)  on  whole 
tissue  sections.  Using  our  system  one  can,  for  instance,  acquire  a  whole- 
view  of  a  section  at  low  magnification  (x2.5,  x10),  and  then  revisit  as 
many  areas  as  he/she  wants  for  acquisition  (e.g.  all  the  ducts  on  a 
section  of  mammary  gland)  at  higher  magnification  (x40,  x63,  x100).  The 
process  is  shown  in  Figure  6.  Multiple  areas  within  a  section  can  be 
background-acquired  by  the  system  without  any  human  interaction. 

Besides  providing  access  to  the  server  functions.  The  second  part  of  the  interface 
(two  left  vertical  panels  In  Figure  2)  expands  the  system  capabilities,  by  allowing  creating 
and  handling  sets  of  related  images,  which  we  call  Cases.  A  case  can  have  multiple- 
color,  multiple-magnification  related  images,  easily  accessible  and  meaningfully  stored. 

For  us,  a  case  is  a  sequence  of  low-magnification  images  of  complete  tissue 
sections  taken  from  a  tissue  block,  along  with  all  the  areas  re-visited  on  them  at  higher 
resolution  and/or  with  different  filters.  Finally  the  user  can  add  information  to  the  cases, 
by  marking  and/or  delineating  structures  of  interest  and  linking  them  within  and  between 
consecutive  sections  (See  Task  1.1).  These  are  the  types  of  marks  and  some  actions 
that  can  be  performed  related  to  them: 

•  Text:  Textual  annotations  to  identify  structures  of  interest  for  future  study  or 

to  draw  the  attention  of  a  different  user. 

•  Duct  Identifies  ductal  structures.  A  section  of  a  duct  perpendicular  to  the 

sectioning  plane  is  marked  with  a  circle  in  the  center  lumen.  Each  circle 
has  a  unique  number  that  identifies  it.  Its  coordinates  are  stored  in  a  data 
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structure  that  contains  information  on  ali  the  ducts  and  their 
interconnections 

•  Connected  Ducts:  Connects  ductal  marks.  Within  the  same  section,  an 

elongated,  longitudinal  section  of  a  duct,  is  normally  marked  with  two 
Duct  marks  connected  by  a  line.  If  both  ends  are  located  on  different 
sections,  the  connection  is  displayed  by  changing  the  coior  of  both  Duct 
marks  when  the  mouse  travels  over  any  one  of  them. 

•  Shapes:  Delineates  irregularly  shaped  ducts  or  other  structures  (e.g.  tumor 

masses,  lymph  nodes,  etc.). 

•  Groups:  Groups  series  or  Shapes  corresponding  to  consecutive  sections. 

The  Shapes  are  grouped  by  sequentially  clicking  on  them  while  the  option 
to  create  Groups  is  on.  To  show  grouping,  the  system  changes  the  color 
of  all  the  Shapes  of  the  group  every  time  the  mouse  moves  into  any  one 
of  them.  Groups  identify  connected  objects,  such  as  sections  of  a  tumor 
mass,  lymph  nodes,  etc. 

•  See  Area:  Displays  previously  acquired  images  of  Areas  of  interst,  when 

the  user  clicks  on  the  rectangle  that  locates  them  within  their 
corresponding  image  section. 

Before  reconstructing  a  Case  in  3D,  all  its  sections  must  be  registered.  For  that, 
we  calculate  the  Rigid-Body  Transform  that  gives  the  optimum  rotation  and  translation 
between  each  pair  of  sections.  The  Transform  is  calculated  from  six  pairs  of  points 
interactiveiy  marked  by  the  user  on  each  pair  of  images  to  be  registered.  Once  the 
points  have  been  marked,  the  software  calculates  the  rotation  and  translation  (9,  tx,  ty) 
needed  to  minimize  the  sum  of  the  squared  distances  between  all  six  pairs  of 
corresponding  points.  The  results  are  stored  in  the  ICS  header  file  of  the  second  image. 
When  registering  a  Case,  all  sections  are  aligned  with  the  first  section  of  the  case,  either 
by  directly  registering  them  to  the  first  section  or  by  using  already  corrected  coordinates 
of  contiguous  sections. 

This  method  is  very  accurate  when  the  pairs  of  points  are  spread  ali  along  the 
sections.  Reasons  for  smali  errors  are  imprecise  mouse  interaction,  stretching  and/or 
compression  of  the  tissue,  and  the  fact  that  some  structures  used  to  select  pairs  of 
corresponding  points  might  not  be  perpendicular  to  the  sections.  Using  as  many  pairs  of 
points  as  possibie  can  average  out  these  small  errors.  That  is  why,  instead  of  using  just 
four  pairs  of  registration  points  -which  is  the  minimum  required  for  this  method  to  work- 
we  use  six. 

An  important  feature  of  R3D2  is  that  all  Case-handling  and  marking  functions  can  be 
used  in  parailel  to  the  functions  that  request  microscope  actions.  Therefore,  acquiring  a 
new  section  can  be  done  in  parallel  to  any  other  Case  related  function  (e.g.  registering 
already  acquired  sections  or  annotating  the  images).  This  is  possible  because  we  are 
using  Solaris  threads.  Threads  permit  executing  multiple  parallel  copies  of  a  program 
without  multiplying  resource  use.  Each  thread  shares  memory  and  other  resources  with 
other  threads.  R3D2  runs  on  a  main  thread  and  when  a  microscope-based  operation  is 
selected,  it  launches  a  new  thread  that  runs  on  the  same  memory  space  as  the  main 
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one.  Thus,  the  client  can  accept  handling  and  marking  operations  on  the  main  thread 
while  a  microscope-based  operation  is  executed  at  the  new  one.  This  also  guarantees 
that,  in  the  case  of  a  microscope  failure  or  a  socket  error,  the  system  will  not  die 
abruptly,  as  only  the  thread  working  on  the  microscope  will  be  affected. 

The  most  innovative  feature  of  our  system  consists  on  its  ability  to  reconstruct  the 
tissue  structures  by  rendering  the  user  markings  in  3D  (See  Task  1.1).  For  that,  we  have 
used  Java3D  (v.1.2.1.)  This  is  an  Application  Programming  Interface  (API)  for  3D  in 
Java.  After  asking  the  user  for  the  range  of  sections  to  render,  the  system  converts  the 
coordinates  of  all  the  markings  in  that  range  of  sections  from  two-dimensional  to  three- 
dimensional  values.  The  section  number  and  thickness,  along  with  the  distance  between 
sections,  determines  the  Z  -depth-  coordinate.  Then  a  3D  scene  is  built  using  several 
geometric  shapes  to  represent  the  different  markings:  spheres  for  sections  of  ducts 
perpendicular  to  the  section,  lines  for  connections  ,  polygons  for  shapes,  ...  Once  the 
Case  is  rendered  (using  another  thread)  it  is  displayed  in  a  new  window  where  the 
mouse  can  be  used  to  rotate  the  scene,  to  zoom  in  or  out  or  to  translate  the  scene  in  the 
X  and/or  Y  directions. 

The  3D  window  includes  a  tool  bar  with  options  to  select  elements.  Selecting 
elements  one  can  get  information  about  them,  load  the  image(s)  that  contain  that 
element  or  move  the  microscope  to  its  location  on  the  slide  for  re-imaging.  That  way,  by 
just  clicking  on  elements  of  the  3D  scene,  the  user  can  both  display  and/or  revisit  areas 
of  interest  which  could  have  been  otherwise  missed  using  a  conventional  2D  analysis 
(See  Task  1.2).  The  window  also  has  buttons  to  hide/show  connections,  shapes,  areas 
(acquired  at  higher  magnification)  and  the  three-dimensional  coordinate  axis.  There  is 
also  a  button  to  reset  the  scene  to  the  default  view  and  combo  boxes  to  the  change  the 
scale  in  any  of  the  three  dimensions. 

We  will  now  illustrate  the  use  of  our  system  by  showing  the  steps  followed  to  image 
and  reconstruct  a  tissue  block  from  a  patient  with  Ductal  Carcinoma  In  Situ  of  the  Breast 
(DCIS).  The  block  was  used  to  test  the  system,  to  determine  the  optimum  spacing 
between  sections. 


Tissue  source 

The  tissue  was  a  Breast  lumpectomy  specimen.  After  surgery  the  tissue  was 
fixed  in  an  alcoholic-formalin  solution,  and  embedded  in  paraffin  per  routine  at  California 
Pacific  Medical  Center  in  1981.  The  case  was  originally  staged  as  a  T1N0M0  (Stage  1). 
Several  blocks  of  approximately  3mm  thickness  were  taken.  The  block  we  used 
contained  DCIS  and  benign  ducts;  there  was  no  invasive  tumor  present.  The  tumor  was 
positive  for  erbB2  by  immunohistochemistry,  performed  a  4  pm  section  taken  from  the 
top  of  the  block. 

Tissue  preparation 

The  block  was  sectioned  at  4  pm  thickness.  We  collected  onto  a  plus  (treated) 
slide  every  5th  section  only.  Every  other  section  collected  was  stained  with  H&E.  The 
rest  of  collected  sections  were  sent  to  the  Cytogenetics  core  for  FISH.  We  collected  66 
tissue  sections,  40  pm  apart.  The  odd-numbered  sections  were  the  H&E  stained 
sections.  The  even-numbered  sections  were  processed  for  FISH,  with  a  probe  for  the 
erb-b2  gene,  which  is  amplified  in  30%  of  carcinomas  of  the  Breast. 
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Imaging  of  tissue  sections 

Full  tissue  sections  were  imaged  in  bright  field  (odd  sections),  or  fluorescence, 
using  a  single-line  filter  block  (360  nm  excitation)  for  DAPI  (even  sections),  using  R3D2’s 
Scan  option  (See  Figure  7).  Sections  were  imaged  at  10X  with  a  Fluar  (0.5  na)  objective 
lens  (Zeiss,  Wetzlar,  Germany).  In  order  to  optimize  memory  use  while  keeping  enough 
resolution  for  Histology,  images  were  reduced  by  a  factor  of  4  in  both  X  and  Y  directions, 
which  gave  as  and  effective  2.5X  magnification,  i.e.  a  sixteen  fold  reduction  in  image 
size.  After  image  compression,  the  average  image  size  of  the  sections  was  25Mbytes. 
The  compression  was  necessary  to  speed  up  image  transmission  from  the  server  and 
display  on  the  client  GUI. 

Creation  of  Cases 

One  case  was  created  initially  empty.  The  acquired  sections  were  added  with  a 
number  equal  to  its  section  number.  Ali  sections  were  manually  registered  to  ensure 
proper  alignment  of  the  to-be-done  annotations.  Each  section  was  registered  to  its 
previous  section,  thus  aligning  ali  sections  with  the  first  H&E  section  of  the  block. 

Tissue  Annotation 

All  H&E  sections  were  manually  annotated.  We  marked  the  centers  of  the  ducts 
by  placing  a  circular  mark  (R3D2’s  Duct  tool)  in  the  lumen  -when  the  duct  was 
perpendicular  to  the  image  plane-  or  with  a  line  —when  the  duct  was  sectioned 
iongitudinally-.  Then  we  connected  the  markings  within  and  between  consecutive 
sections  using  the  Connect  Ducts  option.  We  also  delineated  the  tumor  areas  using 
R3D2’s  Shapes  option  and  grouped  their  consecutive  sections  using  the  Groups  options. 
(See  Figure  8). 

3D  Reconstruction 

The  reconstruction  of  the  case,  based  on  the  H&E  sections  can  be  seen  in  Figure 
9.  All  the  user  markings  are  interactive,  in  that  by  clicking  on  them  the  user  can  i)  obtain 
positional  information,  ii)  load  the  area  of  the  original  section(s)  containing  that  marking 
or  iii)  revisit  the  point  or  area.  For  the  latter  the  microscope  automatically  moves  to  the 
position  of  the  marking  in  the  tissue.  In  situ  tumors  were  rendered  as  volumes  in  3D  and 
ducts  as  lines. 

Revisit  of  Areas  of  Interest 

Several  areas  were  revisited  at  high-resolution  (40X),  with  a  Plan-Neofluar  (1 .3 
na)  oil  immersion  lens  (Zeiss,  Wetzlar,  Germany).  Areas  were  selected  having  either 
morphologically  normal  or  abnormal  (DCIS)  areas.  Areas  were  triple-scanned  using  a 
Pinkel  filter  set  (Chroma  Technologies,  Brattleboro,  VT).  We  consecutively  scanned  the 
sections  exciting  the  sample  at  360,  490  and  572  nm  to  image  DAPI  (nuclei),  FITC  (17c 
probe)  or  CY3  (erb-b2  gene).  The  areas  were  acquired  at  full  resolution  and  then 
compressed  by  a  factor  of  2  in  both  X  and  Y  directions  for  an  effective  20X 
magnification.  Areas  were  manually  selected  from  the  images  of  the  fluorescent 
sections,  although  it  could  have  been  done  from  the  3D  reconstruction  as  well.  To 
ensure  proper  alignment  between  the  areas  and  the  sections,  the  images  of  the  sections 
were  previously  aligned  with  the  microscope  slide  by  calcuiating  the  shift  between  the 
slide’s  current  location  and  its  location  when  the  image  of  the  whole  section  was 
originally  acquired.  Figure  10  shows  the  3D  reconstruction,  now  incorporating  the  high- 
resolution  acquired  areas,  displayed  as  rectangles.  As  all  other  elements  in  the 
reconstruction  modei,  the  areas  can  be  loaded  or  revisited  by  clicking  on  them. 
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Immediate  future  work  will  be  automatically  analyzing  all  the  acquired  areas  to  see  the 
distribution  of  erb2  abnormal  cells  in  different  parts  of  the  block. 

PROBLEMS 


The  main  problems  faced  so  far  are: 

•  The  size  of  the  images,  when  acquiring  entire  tissue  sections  even  at  low 
magnification  (10x),  is  extremely  large  (130  Mb.  or  more).  This  exhausts  the 
computer  memory  when  more  than  one  full-section  is  displayed,  even  using  a 
computer  with  512  Mb  of  RAM  memory,  the  reason  being  the  high  memory 
consumption  of  the  Operating  System  and  the  JAVA  Virtual  Machine,  which 
leave  limited  resources  for  the  client  requirements.  In  order  to  make  the  system 
work  at  a  reasonably  speed  without  compromising  its  use,  we  introduced  a 
compression  factor  which  can  be  set  by  the  user  before  acquiring  a  section.  The 
user  can  choose  between  acquiring  the  image  at  full  resolution  (10X),  half 
resolution  (effective  5x)  or  at  one  forth  of  the  original  resolution  (effective  2.5X). 
Using  the  highest  compression  factor  (%)  is  recommended  when  imaging  tissue 
sections  larger  that  1  cm^.  Smaller  tissue  sections  can  be  acquired  using  half  or 
full  resolution.  Consistency  requires  maintaining  equal  resolution  for  all  sections 
of  a  case.  A  similar  problem  occurs  when  revisiting  areas  at  high-resolution 
(40X).  In  this  case,  we  recommend  using  the  intermediate  compression  factor 
(1/2),  for  an  effective  20X  magnification.  We  are  in  the  process  of  testing  whether 
that  resolution  level  is  enough  for  accurate  FISH  spot  enumeration. 

•  Scanning  whole  sections  is  an  unsupervised  operation,  which  uses  parameters 
previously  defined  by  the  user.  The  exposure  time  is  one  of  these  parameters. 
That  value  is  set  before  starting  scanning  and  does  not  change  until  the  scanning 
is  done.  In  fluorescence,  this  is  a  problem,  since  the  effect  of  the  preparation 
protocols  (e.g.  protein  digestion,  nuclear  permeabilization  in  FISH  or 
Immunohistochemistry,  etc.)  varies  from  cell  type  to  cell  type.  Thus  the 
concentration  of  fluorescent  material  that  reaches  its  target,  may  it  be  the 
chromatin,  a  gene  or  a  protein,  varies  between  different  parts  of  the  tissue. 
Therefore,  the  exposure  time  that  is  appropriate  for  one  part  of  the  tissue  may 
not  be  appropriate  for  another.  That  affects  the  quality  of  the  images,  since  some 
areas  tend  to  be  either  overexposed  or  underexposed.  At  the  moment  we 
manually  selected  an  intermediate  value  that  allows?  us  to  image  dark  areas 
without  overexposing  bright  areas.  This  works  in  almost  all  cases,  but  renders 
images  with  varying  contrast.  To  avoid  that,  we  are  developing  an  algorithm 
which,  when  activated  by  the  user,  will  adjust  the  exposure  time  to  the 
fluorescence  intensity. 

•  An  important  problem  is  that,  due  to  tissue  complexity,  image  annotation  of  real 
tissue  specimens  is  very  time  consuming.  However,  most  of  our  studies 
scheduled  for  the  two  coming  years  don’t  require  fully  sectioning  of  biopsies, 
since  a  correct  planning  the  sectioning  will  reduce  the  number  of  sections 
required.  There  is  also  the  immediate  option  of  increasing  the  distance  between 
sections,  using  only  one  every  N  sections  for  the  reconstruction.  N  has  to  be  low 
enough  as  to  ensure  that  it  is  still  possible  to  determine  ductal  continuity  between 
adjacent  sections.  We  are  now  in  the  process  of  determining  the  optimum  value. 
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Finally,  we  are  working  on  algorithms  to  automate  the  detection  of  ducts.  The 
options  are  to  fully  automatic  detection  and  reconstruction  the  ducts,  followed  by 
interactive  correction,  or  manual  marking  of  an  intermediate  slide  and  then 
automatic  ‘growth’  of  the  structures  from  that  slide  towards  the  contiguous 
sections. 

•  A  final  problem,  related  to  the  previous  one,  is  the  redundancy  of  information  in 
the  3D  reconstruction.  Marking  the  same  structure  on  consecutive  sections 
sometimes  leads  to  3D  objects  with  too  much  level  of  detail.  This  can  complicate 
the  interpretation  of  the  results.  We  are  working  on  the  way  to  ‘prune’  the  3D 
reconstruction  of  the  markings  and  create  and  skeletonized  version  of  it. 
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Task  2.  (Months  1-30)  Using  invasive  cancer  specimens  with  intraductal  extension  of 
DCIS,  identify  DNA  loci  that  are  amplified  in  a  high  proportion  of  the  cells  of  the  invasive 
lesion  using  CGH. 

1.  Select  9  mastectomy  specimens  following  the  cnteria  described  in  the  Methods 
section  of  the  Proposal  body  (Months  1-6) 

2.  Section  and  H&E  stain  mastectomy  specimens  (Months  6-12,  2  specimens; 
Months  12-24.  6  specimens:  Months  24-30.  2  specimens) 

3.  Acquire  sections  using  our  registration  software  (Months  12-24  6  specimens; 
Months  24-30,  3  specimens) 

4.  Reconstruct  the  mammary  ducts  and  identify  the  leading  edge  of  the  intraductal 
component  (Months  12-24,  5  specimens;  Months  24-30,  4  specimens) 

Task  3.  (Months  1-30)  Use  FISH  witfr probes  to  the  loci  identified  by  CGH 

1.  Do  CGH  on  the  selected  mastectomy  specimens  (Months  1-6) 

2.  Do  FISH  to  the  two  most  amplified  regions  and  a  normal  part  of  the  genome  (for 
control  purposes)  on  intermediate  sections  to  those  used  for  the  reconstruction  of 
the  ductal  system  (Months  6-12.  2  specimens:  Months  12-24,  5  specimens; 
Months  24-30.  2  specimens) 

Task  4.  (Months  18-36)  Use  high  magnification  (1  pixeh  0.5  pm)  fluorescence 
microscopy,  look  for  individual  cells  with  the  same  amplified  loci  in  the  ducts 
emanating  from  the  DCIS  lesions.  Assuming  that  we  see  genetically  aberrant  cells, 
measure  the  spatial  relationship  of  these  cells  to  the  surrounding  cells  in  order  to 
characterize  the  pattern  of  aberrant  cells  and  thus  to  provide  information  about  the 
spreading  mechanism  of  the  disease. 

1.  Automatically  enumerate  FISH  spots  and  measure  the  spatial  distribution  of 
aberrant  cells  (if  found)  in  the  histologically  normal  ducts  starting  at  the  very  front 
of  the  intraductal  tumor  expansion.  (Months  18-24,  4  specimens;  Months  24-36, 
5  specimens) 

ACCOMPLISHMENTS 


The  change  on  tissue  source  described  in  the  following  section  (Problems, 
changes),  delayed  the  selection  of  mastectomy  specimens.  Also,  given  the  problems 
associated  to  fluorochrome  fading  with  time,  we  decided  to  select  and  process  each 
case  immediately  before  sectioning  and  staining  it,  instead  of  sectioning  and  staining  the 
tissue  blocks  several  months  before  imaging  the  sections  and  performing  the  analysis, 
which  was  originally  scheduled  to  start  at  month  18"’.  So  far,  we  have  sectioned,  stained, 
imaged  at  low  and  high-resolution  one  case  (described  above).  We  are  now  starting  the 
analysis  of  the  case,  looking  at  the  distribution  of  normal  vs.  abnormal  cells  in  a  Ductal 
Carcinoma  In  Situ  (DCIS)  case. 

PROBLEMS.  CHANGES 

There  has  been  a  change  in  the  tissue  source  and  handling.  Originally,  the  tissue 
that  we  were  going  to  use  was  residual  mastectomy  tissue  provided  by  Dr.  Shanaz 
Dairkee,  from  California  Pacific  Medical  Center  (CPMC).  Scon  after  the  start  date  of  the 
project.  Dr.  Dairkee  informed  us  that,  due  to  the  way  the  tissue  had  been  processed  and 
kept,  the  DNA  of  the  cells  was  not  in  a  state  appropriate  for  FISH.  They  had  tried  several 
times,  with  no  satisfactory  results,  due  to  low  efficiency  of  the  hybridization  and 
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extremely  high  background  autofluorescence,  which  completely  masked  all  the  FISH 
signals. 


It  was  then  suggested  by  Dr.  Dairkee  and  Dr.  Joe  Gray,  who  acts  as  consultant  for 
the  grant,  that  we  use  In  Situ  Hybridization  instead  of  Fluorescence  In  Situ  Hybridization. 
This  required  changing  and  optimizing  the  hybridization  protocol,  since  Dr.  Gray’s  lab 
expertise  is  mainly  FISH.  ISH  was  done  once,  unsuccessfully.  This  confirmed  the  initial 
observation  that  the  DMA  of  the  cells  was  not  in  a  state  appropriate  for  any  type  of 
hybridization.  This,  plus  the  fact  that  moving  from  FISH  to  ISH  would  have  required 
some  important  modification  of  the  microscopy  and  imaging  system,  and  the  fact  that 
ISH  is  not  appropriate  for  multiple  probes  (and  therefore  multiple  colors)  due  to  the 
broad  absorbency  spectrum  of  all  cromogenic  enzymatic  stains,  forced  us  to  look  for  an 
additional  source  of  tissue. 

Dr.  Fred  Waldman,  who  also  acts  as  a  consultant  for  this  grant,  suggested  using 
the  UCSF  tissue  core,  which  collects  tissue  blocks  from  Breast  Cancer  surgery,  being  all 
blocks  appropriately  documented  in  terms  of  histo-pathological  stage.  Everything  else, 
regarding  privacy  on  collecting  and  handling  the  tissue,  remained  the  same,  so  as  to 
ensure  the  validity  of  the  exemption  from  full  Human  Subject  Review  granted  for  the 
original  tissue.  In  any  case,  the  change  was  reported  to  the  Institutional  Review  Board  of 
my  employer  (Lawrence  Berkeley  National  Laboratory).  Once  confirmed  the  approval, 
the  subcontract  with  CPMC  was  voided.  The  sectioning  costs  of  the  tissue  core  were 
substantially  lower  than  the  ones  allocated  under  the  CMPC  subcontract,  and  therefore 
were  included  under  the  existing  subcontract  with  UCSF. 

A  second  change  regarding  tissue  preparation  was  due  to  the  fact  that  after 
submitting  this  grant.  Dr.  Koei  Chin  (Dr.  Gray’s  postdoc  at  the  UCSF  Cancer  Center) 
was  assigned  to  work  on  a  newly  funded  project.  That  prevented  him  from  being  able  to 
do  the  tissue  staining  (CGH  and  FISH)  for  this  project,  as  was  originally  planned  under 
the  subcontract  with  Dr.  Gray’s  lab  at  the  UCSF  Cancer  Center.  Again,  upon  suggestion 
by  Dr.  Waldman,  we  decided  to  make  use  of  the  UCSF  Cancer  Center  Cytogenetics 
core,  which  provides  staining  and  FISH  of  tissue  samples.  The  staining  costs  are 
therefore  paid  using  the  existing  subcontract  without  substantial  budget  change. 
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KEY  RESEARCH  ACCOMPLISHMENTS 


The  first  year  of  the  grant  was  used  to  develop  the  microscopy  and  computer 
infrastructure  that  will  be  used  in  the  following  two  years.  In  summary: 

•  We  have  developed  a  3D  computerized  microscopy  platform  that  allows  image 
acquisition  of  entire  tissue  sections  at  low  resolution,  and  then  provides 
interactive  tools  for  marking  structures  of  interest  (in  our  case  mammary  ducts, 
tumor  masses,  etc.).  The  system  can  reconstruct  the  annotations  in  3D  to 
provide  a  better  understanding  of  tissue  organization  which  can  be  used  to  plan 
high  resolution  analysis  of  areas  of  interest. 

•  The  3D  reconstruction  is  linked  to  the  microscope,  to  allow  revisiting  and  re¬ 
acquisition  of  areas  of  interest  at  high  magnification  and  multiple  fluorescent 
channels  (when  using  fluorescence  microscopy). 

•  To  test  the  system,  we  have  successfully  imaged,  reconstructed  and  revisited  at 
high  resolution  a  biopsy  of  tissue  from  a  patient  with  Ductal  Carcinoma  Is  Situ 
(DCIS)  of  the  breast.  He  tissue  was  fully  sectioned  and  alternatively  stained  with 
H&E  (odd  sections)  and  a  nuclear  fluoresecent  counterstain  (DAPI)  plus  FISH 
with  a  probe  against  the  DNA  locus  of  the  erb-b2  producing  gene  (even 
sections). 
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REPORTABLE  OUTCOMES 


Manuscripts: 

•  “Recent  advances  in  quantitative  digital  image  analysis  and  applications  in  Breast 
Cancer”.  Ortiz  de  Solorzano  C.,  Callahan  D.E.,  Parvin  B.,  Costes  S.,  Barcellos-Hoff, 
M.H.  Review  paper  submitted  to  Microscopy  Research  and  Technique. 

•  “Application  of  three-dimensional  image  analysis  to  the  study  of  Breast  Cancer” . 

Ortiz  de  Solorzano  C.  Proceedings  of  the  XX  Congress  of  the  Spanish  Society  of 
Pathology,  pp.  253-257  (In  Spanish),  Pamplona,  Spain  July  1-5,  2001, 2001. 

•  “A  system  for  computer-based  reconstruction  of  3-dimensional  structures  from  serial 
tissue  sections:  an  application  to  the  study  of  normal  and  neoplastic  mammary  gland 
biology”.  Fernandez-Gonzalez,  R.,  Jones  A.,  Garcia-Rodriguez  E.,  Knowles  D., 
Sudar  D.  Ortiz  de  Solorzano,  C.  Proceedings  of  Microscopy  and  Microanalysis’2001 . 
Microscopy  and  Microanalysis  7(suppl.  2),  pp.  964-5, 2001. 

Presentations: 

•  Application  of  three-dimensional  image  analysis  to  the  study  of  Breast  Cancer.  XX 
Congress  of  the  Spanish  Society  of  Pathology.  Pamplona,  Spain  July  1-5, 2001, 

2001 .  Invited  talk 

•  A  system  for  computer-based  reconstruction  of  3-dimensional  structures  from  serial 
tissue  sections:  an  application  to  the  study  of  normal  and  neoplastic  mammary  gland 
biology.  Microscopy  and  Microanalysis’01,  Long  Beach,  CA  August  5®'-9*^,  2001. 
Platform  presentation. 


Informatics: 

•  As  described  in  the  Body  of  the  report  and  in  the  Reportable  Outcomes  sections,  we 
have  developed  a  three-dimensional  computerized  microscopy  platform,  called 
R3D2. 


Funding  applied: 

•  Segmentation  of  Mammary  Gland  Ductal  Structure  Using  Geometric  Methods.  P.I.’s 
Malladi  R.  and  Ortiz  de  Solorzano  C.  Submitted  to  the  LBNL  Laboratory  Directed 
Research  and  Development  Program  (LDRD)  2002,  in  the  Strategic-Computational 
Sub-Program. 
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Employement  or  Research: 

•  Upon  reception  of  the  Award,  I  was  offered,  and  accepted,  a  Scientist  Position  at  the 
Life  Sciences  Division,  Lawrence  Berkeley  National  Laboratory  of  the  University  of 
California. 

•  The  outstanding  work  of  my  Research  Assistant  (Rodrigo  Fernandez  Gonzalez)  on 
the  development  of  the  microscopy  and  computing  infrastructure  has  been 
instrumental  in  his  acceptance  to  the  joint  UCSF-UC  Berkeley  Ph.D.  program  in 
Biomedical  Engineering.  He  will  continue  working  part-time  on  this  projects  as 
Graduate  Student  Research  Assistant. 
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CONCLUSIONS 


In  summary,  the  goals  for  the  reporting  year  have  been  fully  accomplished.  We 
have  developed  a  three-dimensional  computerized  microscopy  platform  that  allows 
imaging  and  3D-reconstruction  of  tissue  blocks  to  combine  morphological  (Histo- 
Pathology)  and  molecular  (Immunohistochemistry,  FISH)  information.  Our  platform 
directs  and  simplifies  image  acquisition  at  different  resolution  levels  and  rationalizes 
image  storage  and  retrieval. 

Applying  our  system  to  the  test  cases  described  in  this  report  has  helped  us 
understanding  some  practical  limitations  of  the  described  methodology  (e.g.  high 
memory  and  storage  needs  for  the  data  sets  or  high  interactive  burden  when  manually 
annotating  tissue  features).  Practical  solutions  have  been  proposed  and  implemented, 
and  therefore  the  existing  system  is  appropriate  for  the  analysis  planned  for  the  following 
two  years  -studying  the  mechanisms  of  Breast  Cancer  Spread-.  However,  we  will 
devote  some  limited  resources  to  exploring  further  automation  of  the  two  most  time- 
consuming  tasks:  image  acquisition  and  tissue  annotation.  Imaging  of  whole  sections 
can  be  sped  up  by  using  an  automatic  slide  feeder  attached  to  the  microscope,  so  that 
all  sections  can  be  acquired  without  requiring  manual  interaction.  Tissue  annotation 
interaction  can  be  greatly  reduced  by  using  existing  image  analysis  algorithms  to 
automatically  detect  tissue  structures  (Ducts,  lymph  nodes)  and  connect  them  between 
sections.  Interaction  will  be  then  limited  to  annotating  one  section  of  the  case  and 
correcting  the  results  of  the  automatic  detection.  Finally,  our  problems  regarding  tissue 
source  and  preparation  have  been  satisfactorily  solved  as  described  in  the  body  of  this 
report. 


The  relevance  of  what  we  have  achieved  is  significant,  since  our  system  can  and 
will  eventually  be  applied  to  a  variety  of  studies,  both  molecular  and  clinical,  which  will 
greatly  profit  from  the  additional  -third-  dimension.  Several  Pathologists  have  expressed 
their  interest  in  our  system,  arguing  that  it  will  improve  the  way  they  look  at  the  samples 
and  the  way  they  handle  and  store  ‘visual’  information.  Full  use  of  this  system  in  a 
clinical  setting  requires  further  automation  in  the  areas  above  mentioned,  to  provide 
additional  information  without  a  substantial  increase  of  diagnostic  time.  Similarly, 
several  colleagues  in  the  Breast  Cancer  Research  field  have  expressed  their  interest  in 
our  system,  which  would  allow  them  to  do  whole  sample  or  whole  gland  studies  similar 
to  the  one  proposed  in  this  grant. 
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Figure  1.  Description  of  the  system  Figure  1.  Description  of  the  system 


The  application  uses  client-server  architecture.  The  server  runs  on  a 
computer  physically  attached  to  the  microscope,  and  provides  access  to  the 
microscope  functions.  The  client  is  a  JAVA  application,  which  can  run  on  any 
computer  (no  particular  OS  required)  connected  to  the  server  through  the 
Internet  Protocol  (IP).  The  communication  between  client  and  server  uses 
UNIX  sockets.  The  client  provides  user-friendly  access  to  all  the  microscope 
functions  offered  by  the  server,  and  allows  handling  of  sets  of  related  images 
(Cases)  for  storage,  annotation  and  3D  reconstruction  of  structures  of 
interest. 
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Figure  2.  R3D2’s  Graphical  User  Interface. 


The  client  application  (R3D2)  is  a  JAVA  application.  The  GUI  is  divided  in  two  main 
parts.  The  two  leftmost  panels  are  for  displaying  consecutive  sections  of  a  Case.  It 
also  provides  the  user  with  tools  for  registering  the  sections,  annotating  the  images 
and  connecting  structures  between  sections  (e.g.  mammary  ducts,  tumor 
volumes),  and  reconstructing  the  annotated  images  in  3D.  The  rightmost  panel 
gives  access  to  the  microscope  related  functions  offered  by  the  Server. 


Figure  3.  Acquisition  of  a  single  image 

To  acquire  a  singie  image,  the  user  dicks  on  the  “Acquire”  button,  after  choosing 
the  appropriate  microscope  settings:  objective  lens,  excitation  filter  (in 
fluorescence)  and  exposure  time  of  the  CCD  camera.  The  “Autofocus”  option 
automatically  finds  the  correct  focus  plane  of  the  microscope  before  acquiring  an 
image.  The  top  panel  contains  the  entire  image  (reduced  to  fit  in  the  window),  while 
the  lower  panel  contains  a  zoomed  version  of  the  part  of  the  original  image 
selected  by  the  rectangle  on  the  top  window.  The  rectangle  can  be  manually 
moved  to  look  at  different  parts  of  the  image  at  its  original  -full-  resolution. 
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Figure  4.  Panel  for  acquisition  of  whole-sections 


To  acquire  multiple  field-of-view  images,  the  user  is  fills  out  this  form,  which  requests  the 
name  of  the  file  where  the  image  will  be  stored  (if  not  only  displayed),  the  number  of 
scans  -in  fluorescence,  and  which  filter  and  exposure  time  are  to  be  used  in  each  scan, 
and  the  coordinates  of  the  limits  of  the  area.  The  coordinates  can  be  entered  in  the 
boxes,  if  known,  or  otherwise  interactively  defined  at  the  microscope. 


Figure  5.  Whole  section  acquisition 

This  image  shows  the  result  of  scanning  a  complete  tissue  section  from  a  tissue 
block  of  a  patient  with  Ductal  Carcinoma  In  Situ  of  the  Breast  (DCIS),  stained  for 
Histology  with  Hematoxylin  and  Eosin.  As  fo  the  individual  images,  the  upper  panel 
contains  the  entire  section  while  the  lower  contains  a  zoomed  sub-area  of  it. 
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Figure  6.  Revisiting  of  Areas  of  Interest 

Top.  R3D2  showing  a  case  of  a  tissue  biock  from  a  patient  with  Ductal  Carcinoma 
In  Situ  of  the  Breast.  The  block  was  sectioned  and  stained  with  H&E  (odd  sections) 
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and  DAPI  -counterstain-  and  FISH  to  the  erb-b2  gene.  The  interface  shows  two 
sections.  The  left  section  (the  fluorescent  one)  shows  several  light  (yellow) 
rectangles  which  correspond  to  areas  which  were  revisited  and  reacquired  at  a 
higher  resolution.  By  clicking  on  a  rectangle,  the  user  can  display  those  areas  in 
new  panels,  which  show,  in  this  case,  a  counterstained  area  (Bottom,  left)  and  the 
image  channel  corresponding  to  the  emission  of  the  fluorochrome  attached  to  the 
probe  to  the  erb-b2  gene  hybridized  on  the  section  (Bottom,  right). 
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Figure  7.  Tissue  imaging 


The  image  shows  two  sections  from  the  same  DCIS  case  used  to  test  the  systems.  Left. 
Tissue  section  stained  with  H&E  for  standard  histology.  Right.  Tissue  section 
counterstained  with  DAPI.  Microscope  focusing  and  image  acquisition  is  completely 
automated,  as  described  in  the  text. 


Figure  8.  Tissue  Annotation 


Two  consecutive  H&E  stained  sections  of  the  human  DCIS  case.  Only  H&E  sections  are 
now  being  shown.  The  image  show  manual  annotations,  namely  ducts  and  tumor 
masses.  Ducts  are  connected  within  and  between  sections.  Tumor  contours  are  drawn 
and  also  connected  between  consecutive  sections.  Connections  between  sections  are 
shown  by  changing  line  color  of  all  connected  components  every  time  one  of  them  is 
visited  (selected  or  just  traveled  over  by  the  mouse  pointer). 
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Figure  9.  Tissue  Reconstruction. 


3D  reconstruction  of  the  manually  annotated  DCIS  case.  Tumor  volumes  have  been 
surface-rendered  to  show  their  three-dimensional  shapes.  Ducts  are  identified  by 
spheres  and  connected  by  lines  within  and  between  sections.  The  3D  reconstruction  is 
fully  interactive.  It  can  be  rotated,  zoomed,  and  all  the  elements  are  active,  in  that  b  y 
clicking  on  them  one  can  retrieve  information  about  the  element,  display  the  original 
image  that  contains  the  element  or  automatically  move  the  microscope  to  the  selected 
point. 


Figure  1 0.  Tissue  Revisiting.  Different  view  of  the  3D  reconstruction  of  the  DCIS  case, 
with  rectangular  areas  corresponding  to  areas  visited  at  high  resolution.  Those  areas 
can  be  displayed  by  clicking  on  them. 
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